Skip to content

GH-55272: [C++][Parquet][Python] Default to V2 data pages - #55453

Open
Awesome06 wants to merge 6 commits into
apache:mainfrom
Awesome06:gh-55272-parquet-default-v2-pages
Open

Awesome06 wants to merge 6 commits into
apache:mainfrom
Awesome06:gh-55272-parquet-default-v2-pages

Conversation

@Awesome06

@Awesome06 Awesome06 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Parquet V2 data pages have been supported in Apache Arrow for several years, offering key improvements including per-page selective compression (min_compression_savings) and page-level null count headers. This PR switches the default data page format from V1 to V2 across C++ and PyArrow, while retaining full support for writing V1 pages when requested.

What changes are included in this PR?

  • Changes default data_page_version from V1 to V2 in WriterProperties (C++) and PyArrow (parquet.write_table, ParquetWriter, ParquetFileWriteOptions).
  • Refines TypedColumnWriterImpl::WriteArrowDictionary so dictionary is reused when all dictionary entries are referenced even if nulls are present in the chunk.
  • Updates documentation and docstrings in C++ and Python to reflect V2 as the default data page version.
  • Fixes stale comments in type_fwd.h and parquet/core.py describing V2 as an opt-in feature.
  • Updates serialize test assertions and adds regression tests for dictionary statistics handling when nulls are present.

Are these changes tested?

Yes. Existing Parquet C++ and PyArrow test suites cover V2 page writing and reading. New unit tests were added:

  • TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength
  • TestArrowWriteDictionary.NestedDictionaryParentNullStatistics
  • test_min_compression_savings_default_v2

Are there any user-facing changes?

Yes. Parquet files written with default writer properties will now emit V2 data pages rather than V1 data pages. Applications requiring V1 data pages for legacy readers can set data_page_version="1.0".

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow).

V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header.

Closes apache#55272.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 12:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #55272 has been automatically assigned in GitHub to PR creator.

… masking and synchronize V2 page defaults

Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.

Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.

Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.

Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 14:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…ion in TestSerialize and synchronize Sphinx doctest sizes

Update TestSerialize in Parquet C++ to allow selective compression leaving small or incompressible pages uncompressed under V2 page defaults. Wrap comment to adhere to 90-char limit for clang-format.

Synchronize column chunk total_compressed_size and total_uncompressed_size in parquet.rst Sphinx doctest for V2 page defaults.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 17:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…V2 data pages

Update TestSerialize in Parquet C++ to check positivity of compressed and
uncompressed sizes under V2 data pages rather than asserting strict ordering.
Under V2, selective compression can leave data pages uncompressed, while
unconditional compression of dictionary pages can expand small payloads,
making total_compressed_size less than, equal to, or greater than
total_byte_size.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 19:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@adamreeve adamreeve left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @Awesome06! I have a couple of questions

Comment thread cpp/src/parquet/column_writer.cc Resolved
Comment thread cpp/src/parquet/column_writer.cc Resolved
Copilot AI balanced review requested due to automatic review settings November 11, 2026 02:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…atistics with nulls

Add TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength to verify unreferenced dictionary values are not included in statistics when null count causes unique count to equal dictionary length.

Add TestArrowWriteDictionary.NestedDictionaryParentNullStatistics to verify that nested dictionary columns under nullable parents accurately reflect ancestor nulls in page and chunk statistics across both V1 and V2 data pages.
@Awesome06
Awesome06 force-pushed the gh-55272-parquet-default-v2-pages branch from d61267d to fd23951 Compare November 11, 2026 02:45
Copilot AI balanced review requested due to automatic review settings November 11, 2026 02:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 11, 2026 04:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@Awesome06 Please rewrite your PR description as a human writing for humans.

@Awesome06

Copy link
Copy Markdown
Contributor Author

@pitrou Kindly let me know the changes you expect in the PR description. As I'm not quite sure what to reduce or reword in the current description

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@Awesome06 I think my demand was quite self-descriptive. If you are a human, write the PR description yourself instead of pasting AI output.

If you are not a human, please say so and this PR will be closed.

@@ -2024,7 +2024,8 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
::arrow::compute::Unique(*chunk_indices, &exec_ctx));

// On first run, we might be able to re-use the existing dictionary

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update this comment? This isn't about a first run but whether all dict values are referenced.

Suggested change
// On first run, we might be able to re-use the existing dictionary
// Use the original dictionary if we can detect that all indices are referenced,
// otherwise call Take to find referenced dictionary values.

Comment on lines +2027 to +2028
if (chunk_indices->null_count() == 0 &&
referenced_indices.length() == dictionary->length()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to refine the shortcut condition? Something like:

Suggested change
if (chunk_indices->null_count() == 0 &&
referenced_indices.length() == dictionary->length()) {
const bool all_indices_referenced =
(referenced_indices.null_count() == 0)
? referenced_indices.length() == dictionary->length()
: referenced_indices.length() == dictionary->length() + 1;
if (all_indices_referenced) {

} else {
// In Parquet V2, selective compression may keep small or incompressible data
// pages uncompressed (total_compressed_size == total_byte_size), while
// compressed dictionary pages or pages with savings may be larger or smaller

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They shouldn't be "larger or smaller", they should be smaller, precisely because of selective compression.

Comment on lines +1002 to +1003
Using a larger value with V1 or a number outside the allowed range
raises ``ValueError``.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Using a larger value with V1 or a number outside the allowed range
raises ``ValueError``.
Using a larger value with V1 data pages or a number outside
the allowed range raises ``ValueError``.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Default to V2 data pages

6 participants